(1) Field of Invention
The present invention is related to neural networks and, more particularly, to an associative neural network memory endowed with Reinforced Neurogenesis and the ability to indefinitely store new associations without forgetting previously stored information, and without saturating the memory.
(2) Description of Related Art
In machine learning, artificial neural networks are generally presented as systems of interconnected “neurons” which exchange messages between each other. The connections have weights that can be tuned based on experience, making neural nets adaptive to inputs and capable of learning. An associative neural network (ASNN) is a neural network that, using associative memory, includes a function and structure that operate similarly to the correlations in a human brain. An example of such an associative memory is the hyper-dimensional associative memory referred to as Kanerva's Sparse Distributed Memory (SDM) (see the List of Incorporated Literature References, Reference No. 1). Such an associative memory was improved upon Furber et al., in which the SDM was used to store data represented as N-of-M codes for improved storage capacity (see Literature Reference No. 2). Both the SDM and the use of N-of-M codes utilize hyper-dimensional vectors to represent data. Furber's work utilizes sparse data vectors to improve SDM storage capacity, and implements SDM as a bit-matrix representing neural connections for simplicity and high speed of the read/write algorithms. The use of N-of-M codes allows the memory to be (optionally) implemented as biologically plausible spiking neurons, and SDM in general has been identified as a hyper-dimensional model of the human cortex (see Literature Reference No. 3).
Current SDM, with or without the use of N-of-M codes, include several limitations. For example, memory is often limited to a predefined size which is unsuitable for continual storage of new data items over the lifetime of the application. Additionally, statistical correlations in the training data can overload portions of the SDM memory (local saturation) while starving others, resulting premature obsolescence of the storage medium. Further, the more items stored in memory, the worse it performs for classification of incomplete and noisy data. Importantly, there has been little research with regard to indefinite reuse of SDM memory without saturation or dynamic internal load balancing to eliminate premature memory obsolescence. Neural network research in recent decades has yet to produce a truly incremental and robust means of training new information without requiring retraining prior stored information.
Thus, a continuing need exists for an associative neural network memory endowed with the ability to indefinitely store new associations without forgetting previously stored information, and without saturating the memory.